Robustness and Sample Complexity of Model-Based MARL for General-Sum Markov Games

Multi-agent reinforcement learning (MARL) is often modeled using the framework of Markov games (also called stochastic or dynamic games). Most existing literature on MARL concentrates zero-sum but not applicable to general-sum games. It known that best response dynamics in are a contraction. Therefore, different equilibria can have values. Moreover, Q-function sufficient completely characterize equilibrium. Given these challenges, model-based an attractive approach for In this paper, we investigate fundamental question sample complexity algorithms We show two results. first use Hoeffding inequality-based bounds $$\tilde{{\mathcal {O}}}( (1-\gamma )^{-4} \alpha ^{-2})$$ samples per state–action pair obtain $$\alpha $$ -approximate perfect equilibrium with high probability, where $$\gamma discount factor, and {O}}}(\cdot )$$ notation hides logarithmic terms. then Bernstein )^{-1} ^{-2} sufficient. To results, study robustness model approximations. approximate (or perturbed) game always original provide explicit approximation error. illustrate results via numerical example.